A Performance Analysis of Fault Recovery in Stream Processing Frameworks

Distributed stream processing frameworks have gained widespread adoption in the last decade because they abstract away complexity of parallel processing. One their key features is built-in fault tolerance. In this work, we dive deeper into implementation, performance, and efficiency critical feature for four state-of-the-art frameworks. We include established Spark Streaming Flink more novel Structured Kafka Streams test behavior under different types faults settings: master failure with without high-availability setups, driver failures frameworks, worker or exactly-once semantics, application task failures. highlight differences during these on several aspects, e.g., whether there an outage, downtime, recovery time, data loss, duplicate processing, accuracy, cost message delivery guarantees. Our results impact framework design speed explain how use cases may benefit from approaches. Due to task-based scheduling approach, can recover within 30 seconds most necessitating restart. has only a few but slower at catching up delays. Finally, offer end-to-end semantics low requires job restarts leading high times around 50 seconds.